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I .  Obj  ecti ves 


•  Overall  Objective 

•  Effective  collaboration  through  improved  collaborative 
critical  thinking  (CCT) 

•  Objectives  for  this  year 

•  Complete  development  of 

•  CCT  tool 

•  CCT  Training 

•  Collect  validity  evidence  for  both  tool  and  training 
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•  Currently  on-going  at  USF 
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1 1 .  Experiments 


Approach 


•  To  test  the  hypothesis: 

•  CCr  enhances  collaboration  and  perfornnance 

•  We  need  to: 

•  Understand  CCT  at  the  cognitive  and  dispositional  levels 

•  Develop  technology  and  training  that  supports  CCT 

•  Measure  improvennents  in  CCT  and  their  effect  on 
perfornnance 

•  We  will: 

•  Better  understand  CCT 

•  Develop  measures  of  CCT,  and 

•  Strengthen  CCT  w/  tools  and  training 
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Most  Recent  Experinnent 


•  Objective: 

•  To  increase  our  understanding  of  CCT 

•  Determine  the  relative  importance  of  cognitive  and  dispositional 
factors  in  CCT. 

•  Determine  the  impact  on  C2  and  mission  outcomes  of 

•  Training  cognitive  factors  and 

•  Sensitizing  dispositional  factors 

•  Method: 

•  Each  of  3  team  members  receive 

•  Training  in  several  cognitive  aspects, 

•  Sensitization  to  several  dispositional  aspects 

•  Both,  or 

•  Neither 

•  Team  building  exercise 

•  Teams  execute  2  TDGs 
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TDG  #1:  Command  and  Control  Fog 
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•  You  are  commanding  a 
group  of  soldiers  assigned 
to  a  joint  task  force 
conducting  humanitarian 
relief  operations  in  the 
drought  stricken  country  of 
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•  You  are  the  leader  of  the 
Squad  within  the  Platoon 
of  Company  C.  You  are 
fighting  in  a  tropical  area 
against  rebel  forces  armed 
with  handguns,  light 
machineguns,  and  some 
rocket- propel  led  grenades 
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TDG  #2:  Ambush  at  Dusk 
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Some  Details 

•  Pre-  &  post-  task  discussions  for  each  task. 

•  Solution  sheets  and  maps  w/  transparencies  for  each  TDG. 

•  Second  TDG  task  is  video  taped. 

•  Final  questionnaire  for  each  participant. 
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Analyses 


•  Measures 

•  Counts  of  skills  observed  in  dialogues 

•  Self-reported  use  of  skills 

•  Correctness  of  solutions 

•  Goals  (do  they  make  sense?  i.e.,  does  this  violate  tactical 
procedures;  will  they  kill  themselves  by  doing  this) 

•  Orders  (do  they  make  sense?  i.e.,  does  this  violate  tactical 
procedures;  will  they  kill  themselves  by  doing  this) 

•  Map  (does  it  it  portray  what  they  said?  does  it  made  any  sense?) 

•  Analyses  (ongoing) 

•  Evaluate  impact  of  training  &  sensitization  on  outcomes 

•  Estimate  unique  contributions  of  cognitive  and  dispositional 
factors  using  hierarchical  regression 
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Planned  Experiments 
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•  Usability  testing  for  the  tool's  pop-up  feature 

•  Planned  for  early 
February  2004 

•  Many  vs  Few 
probes 

•  Usability  testing  for  the  tool's  "facilitator"  interface 

•  Planned  for  late  February/early  March 

•  Validation  of  the  tool  and  training 

•  Planned  for  March,  April,  May 
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III.  Expected  Final  Products 
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•  CCTTool 

•  CCT  Training 

Potential  impact 

•  Both  process  and  products 

•  I  mproved  collaboration 

•  Better  teann  decisions 

Applications 

•  Any  distributed,  synchronous  team  trying  to  agree  on 
an  optimai  solution. 
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ccr  Tool 


•  The  setting 

•  A  geographically  distributed  team  in  a  long  working  session 

•  The  team  leader  wants  to  monitor  CCT  activity 

•  Team  members  need  reminders  to  engage  in  CCT 

•  Two  components 

•  Participants'  tool  - 

•  Elicits  data  concerning  team  member  monitoring,  assessments, 
critiques,  actions 

•  Cues  team  members  to  monitor,  assess,  critique,  act 

•  Coordinator's  tool 

•  Helps  leader  or  aid  plan,  poll  for,  and  analyze  collaborative  critical 
thinking  activity 

•  Setup  element 

•  Monitoring  elennent 
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Workspaces  in  a  Distributed  Team 


Sign  on  dialog 
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Note;  When  signing  on  to 
the  CCT  Tool  the 
participants  are  already 
logged  onto  the 
collaborative  application. 


Remember;  during  the  work  session  the  critical  thinking 
assessment  tool  will  refer  to  you  as 
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A  Pop-Up  Probe 


Mission  Planning  CoMaborative  Tool 


File  Edit  Window  Help 


^x| 


©J  a&l^li-lQl  ®|(i3 
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Opportunity  to  Rate  and  Comnnent 


jnjxj 


g|  Mission  Planning  Collaborative  Tool 


File  Edit  Window  Help 


gBiAll’lQl 


^^^©0  05:23  PM 


cz? 


b,.CCL  -  Evaluate 


||  This  is  the  context  question  for  the  comment  J-  ' 


Comments: 


Canncel  | 
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Pop-up  "Probe"  Development 
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•  Probes  were  developed  to  elicit  information 
concerning  (at  least) 

•  Whether  the  team  is  engaging  in  CCT 

•  T o  what  topics  it  should  apply  CCT 

•  When  it  should  do  so 

•  I  n  the  probe  development  we  were  mindful  of 

•  3  classes  of  CCT  constructs 

•  CCT  Behaviors 

•  Cognitive  skills  and  dispositions  that  enable  CCT 

•  Phases  of  collaboration 


CCT  objectives 
•  Process 


Products 
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Analysis:  I  dentify  different  suggested 
solutions  or  unmentioned  assumptions 
Evaluation:  Assess  credibility  of  statements, 
compares  strengths/  weaknesses 
Explanation:  J  ustify  one's  reasonings,  write 
clear  plan  of  action 

I  nference:  I  dentify  elements  to  make 
reasonable  conclusions 
I  nterpretation:  Recognize  and  summarize  a 
problem,  organize  info  to  comprehend 
significance 

Self- regulation:  Apply  analysis  and 
evaluation  to  own  judgments 


Monitoring  of  the  need  for  CCT 
Assessment  of  the  time  and  priority 
Critiquing  to  find  information  and  reasoning 
faults 

Action  to  get  information  or  fix  faults 

Dpea; 


It 


he  probe  de 
classes  of 

CCT  Behaviors 


/inTormaiion 


I  nquisitive:  Eager  to  get  knowledge  and  explanations 
J  udicious:  Deliberate  and  careful 
Truthseeking:  Pursues  accurate  and  complete  factual 
knowledge 

Confident  in  reasoning:  Trusts  own  reasoning/ 
critical  thinking  skills 
Open-minded:  Open  to  different  ideas 
Analytical:  Anticipates  consequences;  makes  decisions 
based  on  evidence 

Systematic:  Careful  and  resonable  in  developing 
solution 


Cognitive  skills  and  dispositions  that  enable  CCT 
•  Phases  of  collaboratior 

CCT  objectives 


•  Process 

•  Products 


Establish  conventions 
Develop  shared  understanding 
Develop  collaborative  knowledge 
Consensus  attainnnent 
Validation 
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•  Elicit  a  continuous  numeric  rating 

•  Short 

•  Templated 

•  Time  sensitive 


Form  of  Probes 


•  I  n  your  opinion,  is  the  team's  assessment  of  the  current  situation  correct? 

•  Does  your  team  have  all  of  the  critical  information  needed  to  solve  the 
problem? 

•  Has  the  team  addressed  the  plausible  alternatives  for  solving  the  problem? 

•  I  n  your  opinion,  are  the  team  members  working  toward  the  same  goal? 

•  To  what  extent  have  the  advantages  and  disadvantages  of  the  solution 
been  discussed? 

•  I  n  your  opinion,  have  all  feasible  solutions  been  considered? 

•  How  realistic  is  the  time  line  for  the  plan? 

•  How  appropriately  is  responsibility  allocated  among  team  members? 

•  How  much  are  team  members  communicating  about  the  task  at  hand? 

•  How  successful  have  team  members  been  with  their  roles? 
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Sign  on  dialog  for  coordinator/facilitator 
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Coordinator's  Configuration  Interfaces 


iii.  Collaborator  -  Critical  Thinking  Assessment  Set  Up 


Work  Session  LPf.9b.e?j|  Participants  ] 


x] 


Compose  probe: 

Probe  number  4 

Add  I 

Probe  list: 


Probe  U 

Probe  content 

Presentation 

Time 

Import ...  I 

1 

Probe  number  1 

15 

2 

Probe  number  2 

30 

3 

Probe  numbers 

45 

Edit  I 

Remove 

1 

Probe  presentation  sequence: 


5 

15 

30 

45 

60 

Time  tom  start  > 

1 

2 

3 

Probe  # 

*  To  change  the  presentation  time  of  a  probe  slide  it  to  the  desired  time 


Save 


•  Configure  (clockwise) 

•  Work  session 

•  Probes  &  schedule 

•  Participants 
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Response  Monitoring  Component  -  Polling  Plan  Tab 


Collaborator  -  Monitor  —  Objective:  Develop  ElTective  COA  to  Deal  with  Emerging  SAM  Site 


File  Edit  Tools  Window  Help 


Current  Status 
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{ 

Comments: 

Thre  areX  comments 


1 )  Text  of  the  first  comment 


Polling: 

Next  poll  in:  p7  Min. 


Next  question: _ 

This  is  where  the  text  of  the 
next  question  will  be 
displayed 


Overall  Status 
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Insert 
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Participation  Statistics 
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A  Summary  of  Comments 
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Response  Monitoring  Component  -  Opinion  Tab 
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Probe  1 
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Work- 1  n- Progress;  ideas  for  data  visualization 


Participant  Rating  vs  Group  Average  Rating 
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Work- 1  n- Progress;  ideas  for  data  visualization 


Participant  Comparison 
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Rating  Results  +  Advice 
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•  Python  progrannnning  language. 

•  An  interpreted,  interactive,  object-oriented  programming 
language 

•  Comparable  to  TCL,  Perl,  Scheme,  or  Java. 

•  Can  run  in  any  operating  system. 

•  The  CCr  tool  has  a  client/server  architecture  and  uses  MySQL 
for  the  back-end  database. 

•  The  tool  is  designed  to  run  over  the  I  nternet. 

•  The  users  of  the  tool  do  not  have  to  be  using  the  same 
operating  system  when  the  tool  is  running. 

•  i.e  some  users  can  be  in  Windows  and  some  can  be  in  Linux  and 
still  communicate  and  pass  information  seamlessly. 
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CCT  Training 


Hvmtn 


£nD>T^>*rFni 


•  Define  CCT  and  improve  the  basic  framework  for 
understanding  what  CCT  is 

•  Understand  the  elements  of  CCT 

•  Understand  the  barriers  to  CCT 

•  Facilitate  the  process  of  CCT 
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•  I  nitial  Trainir 

•  Cognitive 

•  Affectiv 

•  10-15  rr 

•  ccr 

•  Control 

•  lOnninuteTc 
solve) 

•  Debriefing 

•  ccr 

•  Control 


•ccr  Training 

•Content 

•ccr  Defined 

A  group  process  in  which  people  work  together 
toward  a  common  goal,  whereby  goal 
accomplishment  requires  an  active  exchange  of  ideas, 
purposeful  self-regulatory  judgment,  reasoned  and 
systematic  consideration  of  evidence, 
counterevidence,  and  context,  in  an  environment  that 
commonly  can  be  characterized  as  uncertain,  or 
where  judgments  are  made  under  uncertainty,  and 
there  is  limited  knowledge  and  time. 

•What  do  you  need  to  do  to  make  it  happen 

•Process  (“hands-on"  task) 

•Content 


•  Simulation/Ti 


•Ensure  shared  understanding  of  CCT  - 
definition  and  process 
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IV.  Planned  Validation 
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"pre-task"  training 
and  Debrief 


Tool? 


ccr 

Control 

Yes 

X 

X 

No 
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X 

I  nitial  Training 
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Training  as  previously  outlined 

Participants  will  then  engage  in 
a  team  task 

•  Team  must  locate  and  rescue  a 
lost  party 

•  Team  must  critically 
collaborate  to  allocate 
resources  and  make  plan 

•  Tool 

•  No  tool 

•  Team  will  then  participate  in  a 
simulated  environment  to 
enact  plan 
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Draft  Experiment 


Hypotheses 

•  HI:  ccr  tools  and/or  training  improve  shared  awareness  of  uncertainty  and  risk. 

( Abi  I  ity/  Process) 

•  H21:  ccr  tools  and/or  training  increase  the  incidence  of  CCT  behaviors. 

(Abi  I  ity/ Process) 

•  H3:  CCT  tools  and/or  training  improve  the  team  pians.  (Products) 

•  H4:  ccr  tools  and/or  training  improve  mission  execution  and  outcomes.  (Effects) 
Materials:  Military  scenario  in  which 

•  some  aspects  of  the  situation  are  well  defined,  others  are  not. 

•  some  risks  can  be  reduced  by  information  gathering  or  probing 

•  some  risks  cannot  be  reduced  and  require  contingency  plans 
Testbed:  Distributed  Dynamic  Decision-making  (DDD)  Simulation 

•  Team  research  testbed 

•  Collaboration  measurement  capability 

•  Developed  at  U.Conn,  freely  available,  used  at  25  labs 
Subjects:  ROTC  and  undergraduate  students 
Method: 

•  Pretest  domain  knowledge  &  critical  thinking  ability 

•  Scenario  (re) planning  phase  execution  phase 

•  Real  time  measures  of  CCT 

•  Posttest  measures  of  CCT 

Analysis:  Multi  level  modeling  supports  analysis  of  group. 


individual,  their  interaction  (individual  on  this  team),  and  error  for  group  and  individual 
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•  CCr  is  an  integral  part  of  the  "Collaboration  Stages  and  Cognitive  Processes" 


m 

Si 


COLLABORATION  AND  KNOWLEDGE  MANAGEMENT  (CKM)  PROGRAM 

STRUCTLHAL  MODEL  OE  TEAM  COLLABORATION 


Problem  Area 
Characteristics 


(MACRO  COGNITR  E  PROCESS  FOCUS) 


Situation 


'  time  pressure 


■  informatioriyktiowiedge 
uncertainty 


'  dynamic  infcrmaticn 


'  large  amount  of  knowiedge 
(cognitive  overload^ 


■  human-agent  interface 
oompleKity 


ream  Types 


'  asynchronous 


■  distributed 


■  culturally  diverse 
'  heterogeneous  kn  owledge 
unique  roles 


command  structure 
(hierarchical  vs.  flat) 


'  rotating  team  members 


Orjeratfojraf  Taslrs 


■  team  decision  making,  COA 


■  develop  shared  understandinr 


•  intelligence  analysis 
(team  data  processing^ 


1 


Cottaboration  Stages  &  Cognitive  Processes 


^leta-Coaaithe: 


j  ijidividiial  cojiveisiiai  of 
I  data  to  fcnoTS-led  Ee 


'  tern  inte^jatioji  ofiiidividjal 
JcnoTRledEe  comnoti  iuid-e^siajidi  jie 


“  team  aEfeement  on  a  common 
solution 


solution  adj  u^stment  to 
5t  Eoals  and  eTLit  criteiia 


Macro-Coaaltive: 


"  individual  mental  model 
constmction 

"  knovled  Ee  intejoperaMity 
development 


■  individual  task,  team  and  domain 
fcnoviledEe  development 

■  individual  kno^sledEe  object 
development 

"  individual  visualization  and 
'epresentation  of  meaning 


"  fcnovled  Ee  intejopej  ability' 
development 

■  iteiative  inSKmation  collection 
and  analysis 

*  team  sJia?ed  undejstandins 
development 

"  develop,  jationalize.  &  visualize 
solution  alternatives 


"  team  neEoiiation  of  solution 
alternatives 


"  convergence  of  individual  mental 
models  to  team  mental  model 


■  team  pattern  leco^itton 

■  team  shared  understanding 
development 

■  converEence  of  individual 
mental  models 

■  critical  thinking 

■  sharinE  hidden  tnowledEe 


"  individiial  tasl:.  team  and  domain 
knoTrt  ledEe  development 
"  develooi  nE  new  hnowledEe 


"  individual  task  knowledge 
development  (continued) 


■  team  aEreement  on  sitmation 


■  team  task  knowledEe 
development  (continued) 


compare  problem  solution 
azainst  Eoals 


team  shared  'iinderstandinE 
development 


converzence  of  individual 
mental  models  of  solution 


“  analyze,  revise  o'ittpirt 


MEchaatsms  for  achleviagMeta,  Macro,  aad  Mkro-Coapiti^  e  Processes  if  applies  to  all  stages) 

*  Verbal  CQinmuiiicaliQiis:  presaitm?  and  discusinj  indhidual  intDrmalion;  disciEsms  team  generated  intormation. 
questioning,  agreeing  disagreeing,  nego  tiating  per^ectives.  dncussing  po  ssible  solutions .  providing  rationale. 

*  QD-Verbal  r n mrmimi c afion s :  facial  espressiom,  luke  clues,  (vocal paralanguageX  band  gestures ^  bo<^^  mo^'ements  (kinesics-X 

touch  (Xi||3iy5;=).persiHial  ^ace.  drav^'ing.  text  messages,  augmented  ^Tdeo.  (cognition  in  objects.). 
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Framework  1:  Collaboration 
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Collaboration  phases* 

1.  Individual  build 
knowledge 

2.  Team  integrates 
individual  knowledge 

3.  Team  negotiates 

solution  4 

4.  Team  tests  &  revises  > 

solutions  I 


*(Letsky  et  al.,  2002) 
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Frannework  1:  Collaboration 
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Collaboration/ 
C2  Effects 


Mission  Effects 
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Framework  2:  Collaborative  Critical  Thinking 


Collaborative  critical  thinking* 
behaviors 

•  Monitoring  for  uncertainty 

•  Detecting  opportunities  to  handle  it 

•  Specifying  problenns 

•  Solving  problenns  &  gathering  info 


Freeman,  et  al.,  2001,  2002;  Cohen,  et 
al.  1997,  1998 
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Framework  3:  Dispositions  Support  Critical  Thinking 


Critical  thinking  skills  may  be 
driven  (in  part)  by  cognitive  skills 
Sl  dispositions*  as  well  as  the 
nature  of  the  task 

Measures 

•  Observational 

•  Standardized  instruments 

•  Self  report 


Facione,  1998 
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Collaborative  Critical  Thinking 
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•  Define,  Measure,  Train  and 
Support  Collaborative 
Critical  Thinking 

•  Measure  its  effects  on  C2  & 
Mission  outcomes 
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VI .  Publications  planned,  technical  contributions 

•  Active  participants  of  the  Transition  Assistance 
Program 

•  Selected  to  participate  in  the  May  Opportunity  Forum 

•  Presented  at  the  I CCRTS  conference 

•  Possible  panel  on  collaboration  tools  for  HFES  2004 

•  Possible  Hawaii  I  nternational  Conference  on  Systems 
Science  -  J  anuary  2005 

•  Possible  Society  for  I  ndustrial  and  Organizational 
Psychologists  2005 
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VI I .  Lessons  Learned 
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•  CCr  is  a  difficult  construct  to  define 

•  "you  know  it  when  you  see  it" 

•  CCT  is  difficult  to  measure  objectively 


University  of 
South  Fiorida 
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Project  Summary 


•  Title:  Collaborative  Critical  Thinking  (CCT) 

•  Jared  Freeman,  Ph.D.,  P.l. 

•  Aptima,  1030  15^^^  Street  NW,  Washington,  DC  20005 

•  202-842-1548  x316 

•  freemanCcDapti  ma.com 

•  Kathleen  Hess,  Ph.D.,  P.M. 

•  Aptima,  12  Gill  St.  Suite  1400,  Woburn,  MA  01801 

•  781-935-3966X219 

•  khessOapti  ma.  com 

•  Objectives 

•  Better  understand  CCT 

•  Develop  validated  training  and  tools  to  improve  CCT 

•  I  mprove  the  process  and  products  of  collaboration  through  improved  CCT 

•  Research  Questions 

•  What  are  the  relative  importance  of  cognitive  and  dispositional  factors  in  CCT? 

•  Can  CCT  behaviors  and  their  effects  be  reliably  measured  in  a  semi-  or  fully-automated  fashion? 

•  Can  we  promote  CCT  behaviors  with  training  and  Job  aids? 

•  Does  improved  CCT  result  in  improved  collaboration? 

•  Project  Status 

•  Experiments  are  on-going  to  1)  better  understand  CCT  and  2)  begin  the  initial  validation  of  the  CCT  tool 
interface. 

•  Development  of  the  tool  and  training  are  well  underway 

•  Validation  studies  are  planned  for  later  in  this  quarter 

•  Preparations  are  on-going  for  the  Navy  Transition  Assistance  Program  May  Opportunity  Forum 
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